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(57) Abstract: A situation 
monitoring device which 
enables monitoring of a variety 
of situations and reporting in 
response to the situation using 
a single device is provided. 
The situation monitoring device 
is easy to install and to use 
and a system therefore can be 
implemented inexpensively. 
The situation monitoring device 
recognizes a place of installation 
where the device is installed 
(step SI 02), holds relational 
information correlating the place 
of installation and the situation 
to be recognized and determines 
a predetermined situation to be 
recognized according to place 
of installation recognition results 
and the relational information 
(step S104), recognizes a 
determined predetermined 
situation (step SI06), and reports 
the result of the predetermined 
situation to a user (step SI 08). 
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DESCRIPTION 

SITUATION MONITORING DEVICE AND SITUATION MONITORING 

SYSTEM 

5 

TECHNICAL FIELD 

This invention relates to a situation monitoring 
device that recognizes a situation of a target object 
and reports that situation^ and a situation monitoring 
10 system in which such situation monitoring device is 
connected to a network, and more particularly, to a 
situation monitoring device and situation monitoring 
system used for monitoring a situation. 

15 BACKGROUND ART 

With advances in continuous internet access and 
expanded broadband service there is a growing awareness 
of security issues, as evidenced recently by the 
commercialization and widespread sale of video 

20 communications equipment for remote monitoring of homes 
and offices. By utilizing these types of existing 
video communications equipment, it is possible to 
construct security systems for observing intrusions by 
suspicious persons and monitoring the weak, such as the 

25 sick, the aged, and children, from a remote location. 
However, with a security system like that 
described above, it is necessary for the user at the 
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remote location to check the video data periodically, 
and thus it is difficult to respond quickly when a 
problem arises. Accordingly, although there is also a 
security system having the ability to detect and report 
5 live objects like the system proposed, for example, by 
Japanese Laid-Open Patent Publication No* 2002-74566, 
such a system provides no more than the ability to 
detect and report the intrusion by a person who might 
be a suspicious person. 

10 In addition, with a security system like that 

described above, due to privacy concerns arising from 
the indiscriminate distribution of video data, the 
situations to which such a system can be adapted are 
limited- In order to solve such problems, a 

15 specialized system has been proposed that does not 
distribute the video itself but instead recognizes 
situations specified by the user and performs 
appropriate processing depending on the situation. 
For example, in Japanese Laid-Open Patent 

20 Publication No. 2002-352354, a system that recognizes 
and reports an emergency situation of a person under 
care, based on information such as response by audio or 
detection of absence by image recognition, is proposed. 
In addition, in Japanese Laid-Open Patent Publication 

25 No. 10-151086, a system that recognizes the situation 
inside the bathroom of the user from video data and 
issues a warning when an emergency is detected is 
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proposed. 

However, all these systems are constructed as 
specialized systems for certain unique situations, and 
are not a single device capable of being adapted to a 
5 variety of situations. Therefore, for example, when 
attempting to construct a security system adapted to a 
plurality of objectives, it is necessary to assemble a 
plurality of specialized devices for handling each and 
every situation, which increases the size and the cost 

10 of the system. Furthermore, these specialized systems 
are difficult to introduce (requiring construction and 
the like) and are not easy to install and use. In 
addition, the composition of a family and the 
situations of its members change over time, making 

15 these types of systems impractical. 

By contrast, with recent advances in image 
processing technology and calculating power, a great 
many devices have been proposed that recognize ordinary 
human movements and situations. For example, in 

20 Japanese Laid-Open Patent Publication No. 6-251159, a 
device that converts feature vector sequences obtained 
from time series images into symbol sequences and 
selects the most plausible from among the object of 
recognition categoriiss based on a hidden Markov model. 

25 In addition, many techniques for recognizing facial 
expression have been proposed, such as the device 
proposed by Japanese Laid-Open Patent Publication No. 
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11-214316 that recognizes such expressions as pain, 
excitement and so forth. 

However, in attempting to achieve an ordinary 
movement/ situation recognition device {that is^ the 
5 capacity to recognize a variety of situations using a 
single device) using these types of techniques, the 
number of mistaken recognitions increases as the 
categories of movement that are the object of 
recognition increase, leading to a further Increase in 

10 the required processing power. 

Furthermore, because these conventional security 
systems report the same generalized emergency target to 
a predetermined reporting destination (such as a 
security firm) whenever any sort of emergency arises, 

15 it is difficult to use the device for multiple purposes. 
For example, in the case of a security system designed 
to monitor a child, it is preferable that the situation 
of the child be reported to the mother. Similarly, in 
the case of a security system designed to monitor 

20 emergencies such as the intrusion of a suspicious 

person or , the outbreeik of a fire , it is preferable that 
the emergency be reported to the security f irro or the 
like quickly. However, it has been difficult to get 
conventional security systems to operate flexibly 

25 according to this sort of wide variety of purposes. 
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DISCLOSURE OF INVENTION 
The present Invention is conceived as a solution 
to the problems of the conventional art, and has as an 
object to provide inexpensively a situation monitoring 
5 device and system configured as a single device that 
that can monitor a variety of situations and report 
depending on the situation, and further, that is easy 
to install and to use. 

To achieve the foregoing object, a monitoring 
10 device according to the present invention has a 

configuration like that described below, that is, a 
situation monitoring device comprising: 

place recognition means for recognizing a place 
of installation where the device is installed; 
^5 information holding means for holding relational 

information relating the place of installation and a 
situation to be recognized; 

determination means for determining a 
predetermined situation to be recognized, in . accordance 
20 with recognition results by the place recognition means 
and the relational information; 

situation recognition means for recognizing the 
predetermined situation determined by the determination 
means ; and 

25 communications means for reporting the 

recognition result of the predetermined situation 
recognized by the situation recognition means to the 
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user. 

In addition, to acliieve the foregoing object, 
another monitoring device according to the present 
invention has a configuration like that described below, 
5 that is, a situation monitoring device comprising: 

situation analyzing means for analyzing a 
situation of a target object; 

discrimination means for identifying a 
predetermined situation from output from the situation 
10 analysis means; 

situation encoding means configured to convert 
the situation into a predetermined signal based on the 
output from the situation analysis means; and 

communications means for reporting the output of 
15 the situation analysis means to the user using the 
situation encoding means. 

According to the present invention, it is 
possible to provide a situation monitoring device and 
system configured as a single device that that can 
20 monitor a variety of situations as well as report 

depending on the situation, and further, that is easy 
to install and to use. 

Other features and advantages of the present 
invention will be apparent from the following 
25 description when taken in conjunction with the 
accompanying drawings, in which like reference 
characters designate the same or similar parts 
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throughout the figures thereof. 

BRIEF DESCRIPTION OF DRAWINGS 
The accompanying drawings, which are 
5 incorporated in and constitute a part of the 

specification, illustrate embodiments of the invention 
and, together with the description, serve to explain 
the principles of the invention. 

FIG. 1 is a flow chart illustrating the flow of 
10 processing performed by a situation monitoring device 
according to a first embodiment of the present 
invention; 

FIG. 2 is a diagram showing the outlines of the 
structure of a situation monitoring system including 
15 the situation monitoring device according to the first 
embodiment of the present invention; 

FIG. 3 is a diagram schematically showing the 
structure of the situation monitoring device according 
to the first embodiment of the present invention; 
20 FIG. 4 is a diagram showing the hardware 

configuration of the situation monitoring device 
according to the first embodiment of the present 
invention; 

FIG. 5 is a diagram showing a control panel of 
25 the controls shown in FIG. 4; 

FIG. 6 is a flow chart illustrating details of 
step S102 shown in FIG. 1; 
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FIG. 7 ±s a diagram schematically showing image 
data obtained in step S602 shown in FIG. 6; 

FIG. 8 is a flow chart illustrating details of 
step S103 shown in FIG. 1; 
5 FIG. 9 is a diagram showing sample display 

contents displayed on an LCD of the controls; 

FIG. 10 is a diagram showing a sample 
recognition information table indicating the relation 
between place of installation, a person who is an 
10 object of recognition and situation recognition 
contents; 

FIG. 11 is a flow chart illustrating details of 
step S104 step shown in FIG* 1; 

FIG. 12 is a diagram showing sample display 
15 contents displayed on the LCD of the controls in step 
S1103 shown in FIG. 11; 

FIG. 13 is a diagram showing the layered 
structure of the software for the situation monitoring - 
device ; 

20 FIG. 14 is a diagram showing a table indicating 

the relation between location code and feature 
parameters ; 

FIGS. ISA, 15B and 15C are diagrams 
schematically showing the structure of a situation 
25 monitoring device according to a second embodiment of 
the present invention; 

FIG. 16 is a flow chart illustrating the flow of 
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processing performed by the situation monitoring device 
according to the second embodiment of the present 
invention; 

FIG. 17 is a diagram showing a sample management 

5 table ; 

FIG. 18 is a flow chart illustrating the flow of 
processing performed by a situation monitoring device 
according to a third embodiment of the present 
invention; 

10 FIG. 19 is a flow chart illustrating details of 

step S1802 shown in FIG. 18; 

FIG. 20 is a diagram showing a sample 
recognition information table indicating the relation 
between a person who is an object of recognition and 
15 situation recognition contents; 

FIG. 21 is a diagram showing hardware 
configuration in a case in which a remote control 
serves as the controls; 

FIG. 22 is a flow chart illustrating the flow of 
20 processing of a situation monitoring device according 
to a third embodiment of the present invention; 

FIG. 23 is a diagram showing the control panel 
of the controls shown in FIG. 4; 

FIG. 24 is a flow chart illustrating details of 
25 a report destination setting process (step S2203); 

FIG. 25 is a diagram showing a sample report 
control information table; 
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FIG, 26 ±s a diagram slxowing sample display 
contents displayed on the LCD of the controls; 

PIG. 27 is a diagram showing a sample display of 
a report destination setting screen displayed on the 
5 LCD of the controls ; 

FIG. 28 is a diagram showing a sample conversion 

table ; 

FIG. 29 is a diagram showing a table indicating 
the relation between location code and feature 
10 parameters ; 

FIG. 30 is a diagram showing the structure of a 
situation monitoring device according to a fourth 
embodiment of the present invention; 

PIG. 31 is a flow chart illustrating details of 
15 a report destination setting process (step S2203); 

FIG. 32 is a diagram showing the contents of the 
report control information table; 

FIG. 33 is a diagram showing an outline of the - 
processing flow of a situation monitoring device 
20 according to a fifth embodiment of the present 
invention? 

FIG. 34 is a diagram showing a sample report 
control information table; 

FIG. 35 is a diagram showing a sample 
25 recognition process software module provided in step 
S2205; 

FIG. 36 is a flow chart illustrating details of 
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the reporting process {S2209); 

FIG. 37 is a flow chart illustrating details of 
the reporting process (S2209); and 

PIG. 38 is a flow chart illustrating details of 
5 the reporting process (S2209). 



BEST MODE FOR CARRYING OUT THE INVENTION 
Preferred embodiments of the present invention 
will now be described in detail in accordance with the 
10 accompanying drawings. 

The situation monitoring device according to the 
present invention recognizes predeteannined situations 
of predetermined target objects in response to the 
installation environment of such device and notifies 
15 the user of a change in situation through a network, 
(First Embodiment) 

FIG. 2 is a diagram showing the outlines of the 
structure of a situation monitoring system, including 
the situation monitoring device according to the first 

20 emboddLment of the present invention. 

In FIG. 2, reference numeral 201 designates a 
situation monitoring device, connected to a network 203 
such as the internet by a line connection device such 
as a cable modem/ ADSL modem 202. Reference numeral 204 

25 designates a portable terminal device such as a 
portable telephone, which receives situation 
recognitions results information that the situation 
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monitoring device 201 transmits. Reference numeral 205 
designates a server device having the ability to 
provide services such as a mail server. 

The situation monitoring device 201 generates a 
5 text docioment showing previously decided, predetermined 
information when predetermined changes in situation 
happen to a target object to be recognized (object of 
recognition) and transmits such information to the mail 
server 205 as an e-mail document in accordance with an 

10 internet protocol. The mail server 205, having 

received the e-mail document, notifies the portable 
terminal device 204 that is the recipient of the e-mail 
transmission in a predetermined protocol that e-mail 
has arrived. The portable terminal device 204 then 

16 accepts the e-mail document held in the mail server 205 
according to the e-mail arrival information. Thus, a 
user in possession of the portable terminal device 204 
can confirm a change in situation of an object of 
recognition that the situation monitoring device 201 

20 detects from a remote location. It should be noted 
that the situation monitoring device 201 may be 
configured so as to have a built-in ability to access 
the network 203 directly, in which case the situation 
monitoring device 201 is connected to the network 203 

25 without going through the in -house line connection 
device 202. In addition, the terminal that receives 
the situation recognition result information is not 
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limited to the portable terminal device 204, and may be 
a personal computer or a PDA (Personal Digital 
Assistance ) , etc . 

FIG. 3 is a diagram showing the outlines of the 
5 structure of the situation monitoring device 201 of the 
first embodiment. In FIG. 3, reference numeral 301 
designates a camera lens that tilts (moves up and down) 
within a frame designated by reference numeral 302. 
Reference numeral 303 designates the outer frame for a 
10 pan movement. The lens 301 pans (moves left and right) 
together with such outer frame. Reference numeral 304 
designates a stand, which contains important units 
other than the camera, including the power supply and 
so forth built in. Consequently, the situation 
15 monitoring device 201 can be made compact and 

lightweight, and moreover, by having a camera that can 
tilt /pan built in, can be easily installed in a variety 
of different locations. 

The user then installs the situation monitoring 
20 device 201 in any location that suits the purpose and 
monitors the situation of a given target object. 

Specifically, the situation monitoring device 
201 can be used in a variety of cases, such as the 
following: 

Placed near infants to confirm their safety. 
Placed near sick persons to confirm their health. 
Placed near the elderly to confirm their safety. 
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Placed at the entrance of a home to confirm the 
coming and going of family members and to monitor the 
intrusion of suspicious persons. 

Placed near windows to monitor the intrusion of 
5 suspicious persons . 

Placed in the bath to confirm the safety of 
occupants. 

The foregoing is a summary description of the 
situation monitoring device according to the present 

10 embodiment and its common uses. Hereinafter, a 
detailed description is given of the processing 
performed by such situation monitoring device, with 
reference to the drawings . 

FIG. 4 is a diagram showing the hardware 

15 configuration of the situation monitoring device 
according to the first embodiment of the present 
invention. In PIG. 4, reference numeral 401 designates 
a CPU (Central Processing Unit), 402 designates a 
bridge, which has the capability to bridge a high-speed 

20 CPU bus 403 and a low-speed system bus 404. In 
addition, the bridge 402 has a built-in memory 
controller function, and the capability to control 
access to a RAM (Random Access Memory) 405 connected to 
the bridge. 

25 A RAM 405 is composed of large -capacity, high- 

speed memories necessary for the operation of the CPU 
401, such as SDHAM (Synchronous DRAM) /DDR (Double Data 
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Rate SDRAM) /RDRAM (Rambus DRAM). In addition, the RAM 
405 is also used as an image data buffer. Furthermore, 
the bridge 402 has a built-in DMAC (Direct Memory 
Access Controller) function that controls data transfer 
5 between devices connected to the system bus 404 and the 
RAM 405. An EEPROM (Electrically Erasable Programmable 
Read-only Memory) 406 stores a variety of setting data 
and instruction data necessary for the operation of the 
CPU 401. It should be noted that the instruction data 
10 is transferred to the RAM 405 during initialization of 
the CPU 401, and thereafter the CPU 401 proceeds with 
processing according to the instruction data in the RAM 
405. 

Reference ntmieral 407 designates a^ RTC (Real 
15 Time Clock) IC, which is a specialized device for 

carrying out time management /calendar management. A 
communications interface 408 is a processor that is 
necessary to connect the in-house line connection 
device (a variety of modems and routers) and the 
20 situation monitoring device 201 of the present 

embodiment, and may for example be a processor for 
processing a wireless LAN 

(IEEE802.11b/IEEE802.11a/lEEE802.11g and the like) 
physical layer and lower layer protocol. The situation 
25 monitoring device 201 of the present embodiment is 
connected to the external network 203 through the 
communications interface 408 and the line connection 
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device 202. Reference numeral 409 designates controls, 
and is a processor that controls a user interface 
between the device and the user. The controls 409 are 
incorporated into a rear surface or the like of the 
5 device stand 304, 

FIG. 5 is a diagram showing a control panel of 
the controls 409 shown in FIG. 4. Reference numeral 
502 designates a LCD that displays messages to the user. 
Reference numerals 503-506 designate buttons for menu 

10 choices, and are used to manipulate the menus displayed 
on the LCD 502. Reference numeral 507, 508 designate 
an OK button and a Cancel button, respectively. The 
user sets the situation to be recognized using the 
control panel 501. 

15 In addition, reference numeral 410 shown in PIG. 

4 designates a video input unit, and includes 
photoelectric conversion devices such as CCD (Charge- 
Coupled Devices ) /CMOS (Complimentary Metal Oxide 
Semiconductor) sensors as well as the driver circuitry 

20 to control such devices, the signal processing 

circuitry to control a variety of image corrections, 
and the electrical cuid mechanical structures for 
implementing pan/tilt mechanisms. Reference numeral 
411 designates a video input interface, which converts 

25 raster image data output from the video input unit 410 
together with sync signals into digital image data and 
buffers it. In addition, the video input interface 411 
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generates signals for controlling tlie video Input unit 
410 pan/ tilt meclianism. 

The digital image data buffered by tbe video 
input interface 411 is forwarded to a specific address 
5 in the RAM 405 using, for example, the DMAC built into 
the bridge 402. Such DMA transfer is, for example, 
activated using the video signal vertical sync signal 
-as a trigger. The CPU 401 then commences processing 
the image data held in the RAM 405 based on a DMA 

10 transfer-completed interrupt signal that the bridge 402 
generates. It should be noted that the situation 
monitoring device 201 also has a power supply, not 
shown. This power supply may, for example, be supplied 
by a rechargeable secondary battery, or, where the 

15 communications interface 408 is a wire LAN, by Power 
Over Ethernet (registered trademark). 

FIG, 1 is a flow chart illustrating the flow of 
processing of the situation monitoring device 201 
according to the first embodiment. This flow chart is 

20 a program loaded into the RAM 405 and processed by the 
CPU 401* 

When the situation monitoring device 201 power 
supply is turned on, in step SlOl a variety of 
initialization processes are carried out. Specifically 
25 described, in step SlOl, an instruction data load (that 
is, a transfer from the EEPROM 406 to the RAM 405)., a 
variety of hardware initialization processes and 
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processes for connecting to the network are executed. 

Tlien, in step S102, a process of recognition of 
the place of installation of such situation monitoring 
device 201 is executed. In the present embodiment, the 
5 installation environment in which such device is 

installed is recognized using video image information 
input by the video input unit 410 . 

FIG- 6 is a flow chart illustrating details of 
step S102 shown in FIG. 1. First, in a step S601, 

10 video data is obtained from the video input unit 410 

and held in the RAM 405. Next, in step S602, the video 
input interface 411 activates the video input unit 410 
pan/tilt mechanism and obtains image data for areas 
outside the area obtained in step S601. FIG. 7 is a 

15 diagrcun showing schematically image data obtained in 
step S602 shown in FIG. 6. The interior of a room is 
sensed over a wide area with the camera image 
acquisition proceeding in the order of A->B->C->D. 

Then, in step S603, it is determined whether or 

20 not the acquisition of image data in step S602 is 

completed. In step S603, if it is determined that the 
acquisition of image data is not completed, processing 
then returns to step S601. By contrast, if in step 
S603 it is determined that the acquisition of image 

25 data is completed, processing then proceeds to step 
S604. 

Then, in step S604, a feature parameter 
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extraction process is performed. It should be noted 
that it is possible to use a variety of techniques 
proposed by the image search algorithm and the like for 
the process of extracting a feature parameter. Here, 
5 for example, the position displacement feature 

extraction method of color histograms, higher-- order 
local auto -correlation features (Nobuyuki Otsu, Takio 
Kurita, Sekita Iwao: "Pattern Recognition", Asakura 
Shoten, pp. 165-181 (1996)) or the like is adopted. 

10 Specifically, feature parameters that use a 

predetermined range of color histogram values and local 
auto-correlation features as features are extracted. 
Moreover, not only these types of primitive features 
may be used but also higher- level feature extraction 

15 methods may be used as well. For example, a technique 
may be used in which a search is made for particular 
objects such as a window, bed, chair or desk (K Yanai, 
K. Deguchi: "Recognition of Indoor Images Using 
Support Relations between Objects", Transactions of the 

20 Institute of Electronics, Information and Communication 
Engineers, vol.J84-DII, no. 8, pp. 1741/1752 (August 
2001)) and the detailed features of those objects 
(their shape, color, etc.) and the special relations 
between the objects are extracted as feature parameters. 

25 Specifically, feature parameters that use the 

presence/position/size/color of the object as features 
are extracted. It should be noted that, in any case. 



wo 2005/119620 



20 



PCT/JP2005/010724 



the feature parameters are extracted from the image 
data held in the RAM 405. 

Then, in step S605, a process of discrimination 
is carried out using the feature parameters obtained in 
5 step S604 and feature parameters corresponding to 

locations already recorded, and a determination is made 
as to whether or not the installation environment is a 
new location in which the device has not been installed 
previously. This determination is carried out with 

10 reference to a table indicating the relation between 
feature parameters and place of installation. 
Specifically, where there exists in the table a place 
of installation having feature parameters in which the 
Euclidean distance is the closest and moreover 

15 exceeding a predetermined threshold, such place of 
installation is recognized as the location where the 
situation monitoring device 201 is placed. It should 
be noted that this determination method is not limited- 
to discrimination by distance, and any of a variety of 

20 techniques conventionally proposed may be used. 

In step S605, if it is determined that the 
installation environment is a new location where the 
device has not been installed previously, processing 
then proceeds to step S606. By contrast, if in step 

25 S605 it is determined that the installation environment 
is a location where the device has been installed 
previously, processing terminates. 
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Then, in step S606, location codes corresponding 
to the feature parameters are registered. FIG, 14 is a 
diagram showing a table indicating the correlation 
between location code and feature parameter. 
5 The "location code" is a number that the device 

manages. When a new place is recognized, an arbitraxY 
number not yet used is newly designated and used 
therefore. The "feature parameter" Pnm is scalar data 
indicating the feature level of a feature m at a 

10 location code n. In the case of a color histogram, for 
example, the Pnm corresponds to a normalized histogram 
value within a predetermined color range. It should be 
noted that, for example, this table is held in the 
EEPROM 406 or the like, 

15 Thus, as described in the foregoing, in step 

S102, the device recognizes the place of installation 
from the image data and generates both a unique 
location code that identifies the place of installation 
and information that determines whether or not that 

20 location is a new location where the device is 
installed. 

Then, in step S103 shown in FIG. 1, the 
situation to be recognized is determined. FIG. 8 is a 
flow chart illustrating details of step S103 shown in 
25 FIG. 1. 

First, in step S801 in FIG. 8, using the results 
of the determination made in step S102, it is 
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determined whether or not the location where the device 
is installed is a new location where the device has 
been installed for the first time. If the results of 
this determination indicate that the location is new, 
5 processing then proceeds to step S802 and the operation 
of setting the object of recognition commences. By 
contrast, if the results of the determination made in 
step S801 indicate that the location is not new, 
processing then proceeds to step S807. 

10 In step S802, the user is prompted, through the 

controls 409, to set the object of recognition, FIG. 9 
is a diagram showing sample display contents displayed 
on the LCD 502 of the controls 409. If it is 
determined that the location is new, then a message 

15 prompting the user to set the object of recognition as 
described in the foregoing is displayed on the LCD 502. 
HThen buttons 504-505 are pressed, previously registered 
persons are displayed in succession. When button 506 • 
is pressed, the person currently displayed is set as 

20 the object of recognition. 

When the selection of the person is completed 
and the OK button 507 is pressed, the person who is the 
object of recognition at the current place of 
installation is set in the table (FXG. 10). It should 

25 be noted that, if a person other than one previously 
registered is selected, then processing proceeds to 
registration of the person who is the object of 
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recognition (905) from a new registration screen (not 
shown). In the registration process (905) shown in FIG, 
9, video of the person to be registered is imaged and 
the feature parameters necessary to recognize such 
5 registered person is extracted from this video data. 
Furthermore, in the registration process (905), the 
user is prompted to enter attribute information for the 
• registered person (such as naime, etc.). 

Fig. 10 is a diagram showing a sample 

10 recognition information table indicating the relation 
between the place of installation, the person who is 
the object of recognition and the contents of the 
situation to be recognized. The location code is a 
unique code assigned to the place recognized in the 

15 place of installation recognition process (step S102). 
The person code is a unique code assigned to a 
previously registered person. It should be noted that 
it is also possible to set a plurality of persons as 
objects of recognition for a given location (as in the 

20 case of location code P0002 shown in FIG. 10). In this 
case, an order of priority of the objects of 
recognition may be added to the recognition information 
table. If an order of priority is set, in the actual 
recognition process step the higher the priority of the 

25 person the more frequently he or she is recognized. 
Furthermore, sometimes a particular person who is an 
object of recognition is not set for a given location 
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(as ±n the case of location code P0003 in PIG, 10), 

Next, in step S803, the object of recognition is 
set. In addition, the device determines that there is 
no change if there is no input for a predetermined 
5 period of time, and in step S804 the actual object of 
recognition is determined. Then, in step S804, the 
recognition information table is checked and the person 
who is the object of recognition is determined. For 
example, if P002 is recognized as the location, then 

10 the' device recognizes the situations of persons HOOOl 
and H0002. It should be noted that, in the case of a 
location for which no particular person is registered 
as the object of recognition, the device recognizes the 
situations of all persons. For example, the device 

15 executes such recognition processes as detection of 
entry of all persons, or detection of all suspicious 
persons . 

By contrast, in step S807, it is determined 
whether or not the place of installation has been 

20 changed. In step S807, if it is determined that the 

place of installation has been changed, processing then 
proceeds to step S805. By contrast, if in step S807 it 
is determined that the place of installation has not 
been changed, processing then proceeds to step S806. 

25 Next, in step S805, through a predetermined user 

interface, the user is notified that there has been a 
change in the place of installation, and furthermore. 
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the recognition information table is checked and the 
persons who are the objects of recognition for the 
place of installation are similarly reported to the 
user. Methods that notify and report to the user 
5 through a display on the LCD 502 of the controls 409 or 
through voice information generated by voice synthesis 
or the like may be used as the user interface that 
notifies and reports to the user. Such processes are 
carried out by the CPU 401. 

10 Next, in step S806, a message concerning whether 

or not to change the contents of the setting is 
displayed for a predetermined period of time on the LCD 
502 of the controls 409, during which time it is 
determined whether or not there has been an instruction 

15 from the user to change the target object. If the 

results of the detezrmination carried out in step S806 
indicate that there has been an instruction to change 
the target object, then processing proceeds to step 
S802 and the object of recognition is selected. By 

20 contrast, if the results of the determination carried 
out in step SB 06 indicate there has not been an 
instruction to change the target object, processing 
then proceeds to step S804. Then, after the object of 
recognition is determined in step S804 described above, 

25 processing terminates. 

Thus, as described in the foregoing, in step 
S102, the situation to be recognized is determined. 
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Once again, a description is given of the process slxown 
in PIG. 1. In step S104 in PIG. 1, the content of the 
situation to be recognized is determined. FIG. 11 is a 
flow chart illustrating details of step S104 shown in 
5 FIG- 1. 

First, in step SllOl, the recognition 
infoirmation table is checked and the person code of the 
person who is the object of recognition is acquired 
from the location code obtained in step S102. In the 
10 example shown in FIG. 10, when the location code P0002 
is recognized, two persons, with person codes HOOOl and 
H0002, are set as the persons who are objects of 
recognition. 

Then, in step S1102, it is determined whether or 
15 not the content of the situation recognition at that 

location has already been set for these persons who are 
objects of recognition. If in step SI 102 it is 
determined that the recognition situation at that 
location has not been set (as in the case of a new 
20 situation), processing then proceeds to step S1103 and 
selection of the content of the situation to be 
recognized is carried out. 

FIG. 12 is a diagram showing sample display 
contents displayed on the LCD 502 of the controls 409 
25 in step S1103 shown in FIG. 11. First, a message 
prompting the user to select the content of the 
situation to be recognized for the designated person is 
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displayed (1201). When buttons 504-505 are pressed, 
preset situation recognition contents are displayed in 
succession. When button 506 is pressed, the content 
currently displayed is set as the situation recognition 
5 content. When selection of the situation recognition 
content is completed and the OK button 507 is pressed, 
the situation recognition content for the person who is 
the object of recognition at the current place of 
installation is set in the recognition information 

10 table (step S1104). It should be noted that, if 

"default" (1202) is set or if there is no input from 
the user after a predetermined period of time has 
elapsed, then the content is automatically set to the 
default. The default is such that a situation 

15 ordinarily set in most cases, such as recognition of 
*room entary and exit" and the like, is automatically 
designated, thereby eliminating the inconvenience 
attendant upon setting. 

By contrast, if the results of the determination 

20 carried out in step SI 102 indicate that the content of 
the situation recognition at that location has already 
been set, then processing proceeds to step S1108 and it 
is determined whether or not there has been a change in 
the person who is the object of recognition. If the 

25 results of this determination indicate that there has 
been in a change in the person who is the object of 
recognition, processing then proceeds to step S1106. 
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By contrast. If the results of the determination 
carried out In step S1108 Indicate there has been no 
change In the person who Is the object of recognition, 
processing then proceeds to step S1107 . 
5 Then, In step S1106, through a predetermined 

user Interface, the user Is notified that a new person 
who Is the object of recognition has been set, and 
furthermore, the recognition Information table Is 
checked and the corresponding situation recognition 

10 content Is similarly reported to the user. Methods 

that notify and report to the user through a display on 
the LCD 502 of the controls 409 or through voice 
information generated by voice synthesis or the like 
may be used as the user interface that notifies and 

15 reports to the user. Such processes are carried out by 
the CPU 401. 

Then, in step S1107, a message concerning 
whether or not to change the contents of the setting is 
displayed for a predetermined period of time, during 

20 which time it is determined whether or not there has 
been an instruction from the user to change the target 
object. If the results of this determination indicate 
that there has been an Instruction to change the target 
object, then processing proceeds to step S1103. By 

25 contrast, if the results of the determination carried 
out in step S1107 indicate that there has not been an 
instruction to change the target object, processing 
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then proceeds to step S1105. 

Then, ±n step S1103 and step S1104, a process of 
setting the situation recognition content is executed 
as with a new setting. If there is no user input after 
5 a predetermined period of time has elapsed, then the 
device determines that there has been no change in the 
contents and in step S1105 determines the content of 
the situation to be actually recognized. Then, in step 
S1105, the recognition information table is checked and 

10 the situation recognition content for the person who is 
the object of recognition is set. 

Thus, as described in the foregoing, by the 
processes of from step S102 to step S104 shown in FIG. 
1, the person who is the object of recognition and the 

15 situation recognition content are determined and the 
actual situation recognition process is executed in 
accordance with the determined conditions. 

Next, in step S105, for example, a major change - 
in the background area of the acquired image data is 

20 detected and it is determined whether or not the place 
of installation of the situation monitoring device has 
been moved. This change in the background area can be 
extracted easily and at low load using difference 
information between f ramies. If the results of the 

25 determination made in step S105 indicate that the place 
of installation has changed, then processing returns to 
step S102 and the place of installation recognition 
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process Is commenced one again. By contrast, if the 
results of the determination made in step S105 indicate 
that the place of installation has not changed, 
processing then proceeds to step S106. Matters are 
5 arranged so that this step S105 is executed only when 
necessary, and thus the processing load can be reduced. 

Next, in step S106 shown in FIG- 1, the person 
decided upon in step S103 is tracked and a 
predetermined situation of such person is recognized. 
10 This tracking process is implemented by controlling the 
pan/tilt mechanism of the camera through the video 
input interface 409. In step S106, for example if 
P0002 is recognized as the location, the device 
executes recognition of the situation, "Have you 
15 fallen?" for the person who is the object of 

recognition HOOOl, and executes recognition of the 
situation, "Have you put something in your mouth?" for 
the person who is the object. of recognition H0002. 
Here, any of the variety of techniques proposed 
20 conventionally can be adapted to that processing 
relating to recognition of the person which is 
necessary to this step (e.g., S. Akamatsu: "Research 
Trends in Face Recognition by Computer", Transactions 
of the Institute of Klectronics, Information and 
25 Communication Engineers, vol. 80 No. 3, pp. 257-266 
(March 1997)). The feature parameters needed to 
identify an individual are extracted during 
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registration as described above. 

In addition, any of the variety of methods 
proposed conventionally can be used for tlxe situation 
recognition technique processed in step S106. For 
5 example, if detecting entry to and exit from a room of 
a particular person or detecting the entry into the 
room of a suspicious person, situation recognition can 
be easily achieved using the results of individual 
identification performed by a face recognition 

10 technique or the like. Moreover, many methods 

concerning such limited situations as feeling ill or 
having fallen have already been proposed (e.g., 
Japanese Laid-open Patent Publication No. 11-214316 and 
Japanese Laid-Open Patent Publication No. 2001-307246). 

15 In addition, a situation in which an infant has 

put a foreign object into his or her mouth also can be 
recognized from recognition of hand movements proposed 
in conventional sign language recognition and the like • 
and from information concerning the position of the 

20 mouth obtained by detection of the face. The software 
that executes the algorithms relating to this process 
of recognition is stored in the EEPROM 406 or the 
server device 205 on the network, and is loaded into 
the RAM 405 prior to commencing the recognition process 

25 (step S106) . 

The software for the situation monitoring device 
201 according to the present embodiment has, for 
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example, a layered structure like that sliown in FIG. 13 
Reference numeral 1301 designates an RTOS (Real TdLme 
Operating System), which processes task management, 
scheduling and so forth. Reference numeral 1302 
5 designates a device driver, which, for example, 

processes device control of the video input interface 
411 or the like. Reference numeral 1303 designates 
middle ware, and processes signals and communications 
protocols relating to the processes performed by the 

10 present embodiment. Reference numeral 1304 designates 
application software. The software necessary for the 
situation recognition processes relating to the present 
embodiment is installed as the middle ware 1303. The 
software with the desired algorithm is dynamically 

15 loaded and unloaded as necessary by a loader program of 
the CPU 401. 

Specifically, when the situation to be 
recognized is determined in step S1105, in the example 
described above two types of processing software models 

20 recognizing the situation "Has person fallen?" for 
person HOOOl and the situation 'Has person put 
something in your mouth?" for person H0002 are loaded 
from the EEPROM 406. By limiting the recognition 
situation by the device installation environment or the 

25 person who is the object of recognition, complication 
of the recognition process algorithm can be avoided and 
a practical system can be built inexpensively. 



wo 2005/119620 



33 



PCT/JP2005/010724 



In addition, it is also possible to provide 
inexpensively a system with even greater expandability 
by storing this type of processing software on another 
server device connected to the network. In this case, 
5 when the content of the situation to be recognized is 
determined (step S1105), the CPU 401 accesses the 
prescribed server device and forwards the prescribed 
software modules from the server device to the RAM 406 
using a communications protocol such as FTP (Pile 

10 Transfer Protocol) or HTTP (Hyper Text Transfer 

Protocol). In step S106 shown in FIG. 1, such software 
is used as situation recognition process software. By 
storing the processing software modules on the server 
device, the capacity of the EEPROM 406 can be reduced, 

15 and moreover, device function expansion (processing 
algorithm expansion) can be easily achieved. 

Then, in step S107 shown in PIG. 1, a 
determination is made as to whether or not the 
predetermined situation had been recognized. If the 

20 results of this determination indicate that such a 

predetermined situation has been recognized, processing 
then proceeds to step S108 and the CPU 401 executes a 
reporting process. This reporting process may, for 
example, be transmitted as character information 

25 through the communications interface 408 according to 
e-mail, instant messaging or some other protocol. At 
this time, in addition to character information, visual 
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Information may be foirwarded as well. In addition, the 
device may be configured so that, if the user is in the 
same house where the device is installed, the user may 
be notified of the occurrence of an emergency through 
6 an audio interface, not shown. 

By contrast, if the results of the determination 
made in step SI 07 indicate that the predetermined 
situation has not been recognized, then processing 
returns to step S105 and a checlc is made to determine 

10 the possibility that the place of installation has been 
moved. If the place of installation has not changed, 
the situation recognition process (step S106) continues. 

Thus, as described above, in the present 
embodiment, in accordance with the results of the 

15 recognition of the place of installation the situation 
monitoring device, the situation to be recognized and 
the person who is to be the object of recognition are 
determined automatically, and furthermore, the 
appropriate recognition situation is set automatically 

20 in accordance with the results of the recognition of 
the person who is the object of recognition. 
Consequently, it becomes possible to implement an 
inexpensive situation monitoring device that uses few 
resources. In addition, merely by placing the device 

25 in an arbitrary location, a situation monitoring 

capability can be provided that is suitable for that 
location, and since a single device handles a variety 



wo 2005/119620 



35 



PCT/JP2005/010724 



of Situations ±t is convenient and simple to use, 
(Second Embodiment) 

FIGS. ISA, 15B and 15C are diagrams 
schematically showing the structure of a situation 
5 monitoring device according to a second embodiment of 
the present invention. Reference numeral 1501 shown in 
FIG. 15A designates the main part of the situation 
monitoring device, containing the structure shown in 
the first embodiment. Reference numerals 1502a-1502c 

10 shown in PIGS, 15A-15C designate a stand called a 

cradle, with the main part set in the cradle. To the 
main part 1501 is attached an interface for supplying 
power from the cradle 1502 and an interface for 
inputting information. The cradle 1502 is equipped 

15 with a device that holds information for uniquely 
identifying the power supply and the cradle. An 
inexpensive information recording device such as a 
serial ROM can be used as that device, and can 
communicate with the main part 1501 through a serial 

20 interface • 

The processing operation performed by the 
situation monitoring device of the second embodiment 
differs from the processing operation performed by the 
first embodiment only in the process of step S102 shown 

25 in FIG. 1. 

FIG. 16 is a flow chart illustrating the flow of 
processing performed by the situation monitoring device 
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according to the second embodiment. 

First, in a step S1601, the CPU 401 accesses the 
serial ROM built into the cradle 1502 through a serial 
interface, not shown, and reads out ID data recorded on 
5 the ROM. Here, the read-out ID code is a unique code 
that specifies the place of installation. Then, in 
step S1602, a table that manages the ID code is checked. 

Then, in step S1603, it is determined whether or 
not the place of installation of that ID code is a new 

10 location. It should be noted that the management table 
is assumed to be stored in the EEPROM 406. FIG. 17 is 
a diagram showing a sample management table, in which 
ID codes corresponding to arbitrary location codes that 
the situation monitoring device manages aire recorded. 

15 If the results of the detextnination made in step S1603 
indicate that the place of installation of the ID code 
is a new location, then processing proceeds to step 
S1604 and that ID code is recorded in the management 
table in the EEPROM 406. By contrast, if the results 

20 of the determination made in step S1603 indicate that 
the place of installation of the ID code is not a new 
location, processing then proceeds to step S1604. 

In the case of the present embodiment, by 
setting the main part 1501 on the cradle 1502, the 

25 cradle so set is recognized, and consequently, the 

location where the device is installed is recognized. 
It should be noted that the processing steps that 
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follow the place of installation recognition process 
(step S102) are the same as those of the first 
embodiment, with the object of recognition and the 
situation to be recognized determined according to the 
5 location. 

In addition, in the case of the present 
embodiment, the user installs in advance cradles in a 
plurality of locations where the situation monitoring 
device is to be used and moves only the main part 1501 

10 according to the purpose for which the device is to be 
used. For example, cradle 1502a is placed in the 
entrance hallway and cradle 1502b is placed in the 
children's room. Accordingly, if, for example, the 
main part 1501 is set on the cradle 1502a, the device 

15 operates in a situation recognition mode that monitors 
for entry by suspicious persons, and if set on the 
cradle 1502b, the device operates in a situation 
recognition mode that monitors the safety of the 
children • 

20 As is clear from the foregoing description, 

according to the second embodiment, the place of 
installation can be recognized accurately by using a 
simple method in which the location is recognized by 
acquiring an ID code. 

25 (Third Embodiment) 

FIG. 18 is a flow chart illustrating the flow of 
processing performed by a situation monitoring device 
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according to a tliird embodiment of the present 
invention. The flow chart Is a program loaded into the 
RAM 405, and processed by the CPU 401. In the case of 
the present embodiment as well, the hardware 
5 configuration is the same as that of the first 
embodiment of the present invention, and thus a 
description is given of only that which is different 
from the first embodiment. 

When the power to the situation monitoring 

10 device is turned on, in step S1801 a variety of 

initialization processes are executed. Specifically, 
in step S1801, processes are executed for loading 
instruction data (forwarding data from the EEPROM 406 
to the RAM 405), initialization of hardware, and 

15 network connection. 

Then, in step S1802, the content of the object 
of recognition and the situation to be recognized for 
that object of recognition are selected. FIG. 19 is a 
flow chart illustrating details of step S1802. 

20 In step S1901, the user is prompted to set the 

object of recognition through the controls 409. PIG. 9 
is a diagram showing sample display contents displayed 
on the LCD 502 of the controls 409. First, a message 
prompting the user to select an object of recognition 

25 is displayed (901). When buttons 504-505 are pressed, 
previously registered persons are displayed in 
succession. When button 506 is pressed, the person 
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currently displayed is set as the object of recognition. 

When the selection of the person is completed 
and the OK button 507 is pressed, the person who is to 
be the object of recognition at the current place of 
5 installation is recorded in the table (step S1902). It 
should be noted that, if a person other than one 
previously registered is selected, then, as with the 
first embodiment, the device enters a mode of 
registering the person who is to be the object of 

10 recognition from the new registration screen 905. 
FIG. 20 is a diagram showing a sample 
recognition information table showing the relation 
between a person who is the object of recognition and a 
situation to be recognized. 

15 The codes for the person who is the object of 

recognition are unique codes assigned to previously 
registered persons. In addition, codes having a 
special meaning can be assigned to the person who is 
the object of recognition. For example, in the example 

20 shown in FIG. 20, H9999 is a special code indicating 
that all persons are targeted. When such a code is 
selected, a predetermined situation is recognized for 
all persons. 

Then, in a step S1903, the type of person 

25 selected as the object of recognition as well as the 

situation recognition content are reported to the user. 
Methods that notify and report to the user through a 



wo 2005/119620 



40 



PCT/JP2005/010724 



display on tlie LCD 502 of the controls 409 or througli 
voice information generated by voice synthesis or the 
like may be used as the user interface that notifies 
and reports to the user. 
5 In step S1905, a display querying the user 

whether or not the selected content of the situation 
recognition is to be changed is carried out for a 
predetermined period of time, and a determination is 
made as to whether or not there has been an instruction 

10 from the user to change the selected content of the 

situation recognition within the predetermined period 
of time. If the results of this determination indicate 
that there has been an instruction from the user to 
change the selected content of the situation 

15 recognition, processing then proceeds to step S1906. 
By contrast, if the results of that determination 
indicate that there has been no instruction from the 
user to change the selected content of the situation 
recognition, then processing terminates, 

20 Then, in step S1906, the content of the 

situation to be recognized for each person who is the 
object of recognition is set. For example, when the 
buttons 504-505 are pressed, preset situation 
recognition contents are displayed in succession. When 

25 button 506 is pressed, the content currently displayed 
is set as the situation recognition content. When 
selection of the situation recognition content is 
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completed and the OK button 507 is pressed, the 
situation recognition content for the person who is the 
object of recognition at the current place of 
installation is set in the recognition information 
5 table (step S1104). It should be noted that, if 

"default" (1202) is set or if there is no input from 
the user after a predetermined period of time has 
elapsed, then the content is automatically set to the 
default. The default is such that a situation 

10 ordinarily set in most cases, such as recognition of 
"room entry and exit" and the like, is automatically 
designated, thereby eliminating the inconvenience 
attendant upon setting. 

HThen setting of the situation recognition 

15 content is completed, the actual recognition operation 
is commenced. First, in step S1803 shown in FIG. 18, 
the process of detecting and recognizing the object of 
recognition is carried out. Here, too, as described 
with respect to the first embodiment, any 

20 conventionally proposed person recognition algorithm or 
the like can be used for the process of recognizing the 
target object. It should be noted that if the person 
detected is a new person not set in the recognition 
information table, then the process of setting the 

25 person in the recognition information table is carried 
out in the setting step (S1802)* However, in step 
S1804 the determination whether or not to move to the 
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setting process can be set in advance by the user. 
That is, when a person not set in the table is detected, 
it is also possible to set the device to routinely 
ignore that person or carry out previously determined 
5 default situation recognition. 

Then, in step S1805, the recognition information 
table is checked and the situation recognition content 
for the recognized person is determined- Then, in step 
S1806, the situation recognition process for the 

10 situation recognition content determined in step S1805 
is executed. As with the first embodiment, the 
situation recognition performed here can also be 
accomplished using any of the variety of methods 
proposed conventionally. Then, in step S1807, when it 

15 is determined that a predetermined recognition of a 
predetermined person has been identified, as with the 
first embodiment, in step S1808, the user is notified. 

Thus, as described above, with the third 
embodiment, the situation to be recognized is 

20 automatically determined for each person who is the 
object of recognition and an appropriate situation 
recognition is automatically set. Consequently, it is 
possible to implement an inexpensive system that uses 
few device resources. In addition, merely by placing 

25 the device in an arbitrary location, a situation 

monitoring capability can be provided that is suitable 
for that location, and since a single device handles a 
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variety of situations It Is convenient and simple to 
use. 

It should be noted that, although the foregoing 
embodiments are described In terms of a person who Is 
5 the object of recognition, the present invention is not 
limited to such a situation and may, for example, be 
adapted to any object of recognition, such as an animal 
or a particular object, etc. For example, in the case 
of a particular object, the device may be used to 

10 recognize and report such situations as that such 

object "has been moved from a predetermined position" 
or "has gone missing" . Recognition of movement or 
presence can be accomplished easily by using a pattern 
matching technique, proposed conventionally, 

15 In addition, although the foregoing embodiments 

are described in terms of recognizing the location 
where the device is installed and the situation of the 
object of recognition target using video Information, 
the present invention is not limited thereto and may, 

20 for example, be configured so as to recognize situation 
using sensing Information other than video information. 
Furthermore, the present invention may use a 
combination of video information and other sensing 
information. Information gathered by voice, infrared, 

25 electromagnetic wave or other such sensing technologies 
can be used as the sensing information. 

In addition, although the foregoing embodiments 
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are described In terms of defining the relation between 
the place of installation, the object of recognition 
and the situation recognition content using an ordinary 
table, the present invention is not limited thereto and 
5 may, for example, make determinations using higher 
level recognition technologies. For example, a 
technique may be used in which high-level 
discrimination is carried out concerning the 
significance of a location (i.e., that the place is a 

10 child's room or a room in which a sick person is 

sleeping) from the recognition of particular objects 
present at the place of installation or the 
identification of persons appearing at such location, 
and using the results of such recognition and 

15 identification to determine the object of recognition 
and the situation recognition content. 

In addition, although the first embodiment 
described above is described in terms of commencing the 
process of recognition of the place of installation of 

20 the device using a change in the acquired background, 
the present invention is not limited thereto and may, 
for example, use other techniques. For example, a 
method may be used in which, a mechanical or an optical 
sensor is attached to the bottom of the device that 

26 detects when such device is picked up and later set 

down again, with location recognition commenced at such 
times. Moreover, a method may be used in which the 
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process of recognizing the location is commenced when a 
predetermined button on the controls is set. In either 
case, the processing load can be reduced compared to 
executing the location recognition process continuously. 
5 Furthermore, a method like that in which the location 
recognition process is commenced automatically at 
predetermined time intervals using the RTC 407 may be 
used. In this case as well, the processing load can be 
reduced compared to executing the location recognition 

10 process continuously. 

In addition, although the second embodiment 
described above is described in terms of recognizing 
the place of installation by the different cradles on 
which the situation monitoring device is set, the 

15 present invention is not limited thereto and may, for 
example, use other techniques. For example, the device 
may be given a built-in wireless tag receiver so that, 
for example, the place of installation of the device 
may be detected by detecting a wireless tag affixed at 

20 a predetermined location within the house. In this 

case, the wireless tag can be provided by a seal or the 
like, thus making it possible to implement, easily and 
inexpensively, a reliable place of installation 
detection capability. Furthermore, the device may be 

25 given a built-in, independent position information 

acquisition unit in the form of a GPS (Global Position 
System) or the like, and the information obtained by 
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such unit used to acquire the position of the device 
inside the house, etc. In this case, by combining GPS 
position detection results and image detection results, 
it is possible to provide a more accurate place of 
5 installation recognition capability. 

In addition, although the foregoing embodiments 
are described in terms of using internet e-mail as a 
medium of reporting a change in the situation of the 
object of recognition, it is conceivable that problems 

10 might occur with real-time transmission if e-mail 

protocols are used. Accordingly, other protocols may 
be used. For example, by using instant messaging 
protocol and the like, it is possible to achieve rapid 
information reporting. Moreover, the invention may be 

15 configured so that, instead of reporting by text 

message, the device main unit is provided with a built- 
in telephone capability and voice synthesis capability, 
so as to contact the remote location directly by 
telephone to report the information. 

20 In addition, although the foregoing embodiments 

are described in terms of using a camera having a 
mechanical control structure (a so-called pan/tilt 
camera), the present invention is not limited thereto 
and may, for example, employ a wide-angle camera 

25 instead. In that case, the object of recognition is 

not supplemented mechanically but instead an equivalent 
process can be implemented using image data acquired at 
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wide angles. 

In addition, although the foregoing embodiments 
are described In terms of providing the device main 
unit with a control unit having an Input /output 
5 capability as the controls, the present invention is 
not limited thereto and may, for example, employ a 
remote control or the like that Is separate from the 
device as the control unit. FIG. 21 is a diagram 
showing the hardware configuration in a case in which a 

10 remote control is used for the control unit. In FIG. 
21, only the controls 2109 are different from the 
hardware configuration described with respect to the 
first embodiment above (PIG. 4). Thus, reference 
numerals 2109b, c designate communications units for 

15 controlling communications between the controls I/F 
2109 and the main unit. Implemented using a wireless 
interface such as an electromagnetic wave or Infrared 
wireless interface. These communications units can be • 
Implemented easily and Inexpensively using low- speed 

20 wireless transmission medium. Reference numeral 2109a 
designates the controls I/P, which is equipped with 
display/input functions like the controls 409 shown in 
the first embodiment. A remote control 2109d, 
consisting of the controls I/F 2109a and the 

25 communications unit 2109b, is lightweight and compact. 
The user can set parameters needed for the operation of 
the device by operating the remote control 2109d. 
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Sepsiratlng the controls from the main unit in the 
foregoing manner provides greater flexibility in the 
installation of the device and enhances its convenience 
as well. 

5 Furthermore, the invention may be configured to 

set the parameters needed for operation using a network. 
For example, the invention may be provided with an HTTP 
(Hyper Text Transfer Protocol) server capability and 
the user provided with a Web-based user interface based 

10 on HTTP via a communications interface 2108. The HTTP 
server may be incorporated as one part of the middle 
ware (reference numeral 1303 shown in FIG. 13), 
activating a predetermined parameter setting program in 
response to input from the remote location based on 

15 HTTP. The user is able to set the parameters needed 
for operation of the main unit from an ordinairy 
terminal such as a mobile telephone, a PDA, a personal 
computer or the like. Furthermore, such setting 
operation can be carried out from the remote location. 

20 Moreover, the device can be implemented inexpensively 
because it does not require provision of a special 
control unit. 

In addition, although the foregoing embodiments 
are described in terms of executing all processes using 

25 a processor incorporated in and built into the 

situation monitoring device, the present invention is 
not limited thereto and may, for example, be 
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Implemented In combination with a personal computer or 
otlier such external processing device. In that case, 
only the reading In of Image data Is accomplished using 
a special device, with all other processing, such as 
5 image recognition, communications and so forth, 

accomplished using personal computer resources. By 
using a wireless Interface such as BlueTooth, for 
example, or a power line communications Interface such 
as HPA (Home Power Plug Alliance) or the like to 

10 connect the specialized device and the personal 

computer, the same convenience as described above can 
be achieved. This sort of functionally dispersed 
situation monitoring system can of course be achieved 
not only with the use of a personal computer but also 

15 with the aid of a variety of other Internet appliances 
as well* 

In addition, although the foregoing embodiments 
are described in terms of implementing the present 
invention by software processing using a CPU, the 

20 present invention is not limited thereto and may, for 
example, be Implemented by special hardware processing 
as well. In that case, the algorithm for situation 
recognition corresponds to object data that determines 
the internal circuitry of an FPGA (Piled Programmable 

25 Gate Array) or object data that determines the internal 
circuitry of a reconf Igurable processor. When the 
situation to be recognized Is determined (step S1105), 
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the system control processor loads the data from the 
EEPROM 406 or a server device connected to the network 
or the like into the special hardware. The special 
hardware then commences recognition processing of a 
5 predetermined algorithm according to the object data 
that has been loaded. 

Thus, as described above, according to the 
present embodiments, because the content of the 
situation to be recognized is limited depending on the 

10 place of installation of the device itself, it is 
possible to achieve a reliable situation monitoring 
device inexpensively. Moreover, because the place of 
installation is diagnosed automatically and the 
appropriate situation to be recognized is determined 

15 accordingly, the user can recognize a variety of 
situations simply by installing a single device. 

In addition, according to the above -described 
embodiments, because the object of recognition and the 
situation recognition content are limited according to 

20 the place of installation of the device, it is possible 
to achieve a more reliable situation monitoring device 
inexpensively. Moreover, because the place of 
installation is diagnosed automatically and the 
appropriate object of recognition and situation to be 

25 recognized are determined accordingly, the user can 
recognize a desired situation with a high degree of 
reliability simply by installing the device. 
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In addition, according to the above- described 
embodiments, because the situation recognition content 
is limited according to the object of recognition, it 
is possible to achieve a reliable situation monitoring 
5 device inexpensively. Moreover, the user can recognize 
a desired situation simply by placing the device near 
the target object of recognition or a location where 
there is a strong possibility that the target object of 
recognition will appear. 

10 In addition, according to the above -described 

embodiments, the device can be implemented 
inexpensively without the need for special sensors and 
the like. Moreover, carrying out location recognition 
processing only where necessary enables the processing 

15 load to be reduced. As a result, location recognition 
processing can be commenced reliably with an even 
simpler method. Furthermore, location recognition 
processing can be commenced reliably without the 
addition of special sensors and the like. 

20 Moreover, it is possible to prevent errors in 

the recognition function produced by erroneous 
recognition of the place of installation. It is also 
possible to prevent errors in the recognition function 
produced by erroneous recognition of the object of 

25 recognition. It is also possible to provide a user 
interface for setting information at the appropriate 
time, thus improving convenience. 
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In addition, according to the above -described 
embodiments, because It Is possible to provide a user 
Interface for setting Information automatically wlien 
changing the place of Installation, thus improving 
5 convenience- It Is also possible to provide a user 
Interface for setting information only when changing 
the place of Installation, and even then only when 
necessary, thus Improving convenience* It is also 
possible to provide a user interface for setting 

10 information only when necessary, depending on the 

results of the recognition of the object of recognition 

In addition, according to the above-described 
embodiments, providing a user interface for setting 
information only when necessary Improves convenience 

15 and makes it possible to achieve more desirable 
situation recognition depending on the order of 
priority. It is also possible to recognize the place 
of installation of the device reliably using a simple 
method. 

20 In addition, the above-described embodiments 

make It more convenient for the user to set the 
parameters necessary for operation of the device, and 
also enable the user to set the parameters necessary 
for the operation of the device from a remote location. 

25 It is also possible to set the parameters necessary for 
the operation of the device from an ordinary terminal. 
In addition, it is possible to achieve a more 
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compatible device with greater expansion capability 
inexpensively . 

(Fourth Embodiment) 

FIG. 22 is a diagram showing the outlines of a 
5 processing flow performed by a situation monitoring 
device according to a fourth embodiment of the present 
invention • Such processing flow is a program loaded in 
the RAM 405 and processed by the CPU 401. 

When the situation monitoring device 201 power 

10 supply is turned on, in step S2201 a variety of 

initialization processes are carried out. Specifically, 
instruction data load (that is, a transfer from the 
EEPROM 406 to the RAM 405), hardware initialization and 
connection to the network are executed. 

15 Next, in a step S2202, a process of identifying 

the place of installation is executed. In the present 
embodiment, the place of installation of the device is 
identified using video image information input using 
the video input unit 410. It should be noted that the 

20 details of the place of installation identification 

process (step S2202) are the same as those described in 
FIG. 6 with respect to the first embodiment described 
above, and thus a description thereof is omitted here 
(the table indicating the relation between the location 

25 codes and the feature parameters are the same as in FIG. 
14 (see FIG. 29) ) . 

Alternatively, instead of performing the 
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identification of the place of installation 
automatically, the device may be configured so that the 
user performs this task manually- In that case, the 
user inputs information designating the place of 
5 installation through an interface, not shown, displayed 
on the control panel 501 of the controls 409. 

In addition, when selecting the destination for 
the reporting of the situation recognition content or 
the reporting medium, when not using information 

10 relating to the place of installation, the place of 

installation identification process (step S2202) or the 
place setting process may be eliminated. 

Next, in step S2203, the destination of the 
reporting when a predetermined situation is recognized 

15 is set. FIG. 24 is a flow chart illustrating details 
of a report destination setting process (step S2203). 

In step S2401, an interface, not shown, querying 
the user whether or not to change the settings is 
displayed on the control panel 501 of the controls 409. 

20 In the event that the user does change the settings, 
the setting information stipulating the reporting 
destination is updated in the steps {S2402-S2405) 
described below. 

First, in step S2402, the user is prompted to 

25 set the object of recognition through the controls 409 
(reference numeral 901 in FIG. 9). It should be noted 
that FIG. 9 shows sample display contents displayed on 
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the LCD 2301 (FIG. 23) of the controls 409, 

Here, when buttons 504-505 are pressed, 
previously registered persons are displayed in 
succession (902-904). When button 506 is pressed, the 
5 person currently displayed is set as the target of a 
reporting event occurrence. When selection of the 
situation recognition content is completed and the OK 
button 507 is pressed, the person who is the object of 
recognition at the current place of installation is set 

10 in a reporting control information table (FIG. 25). 

The reporting control information table is table 
data stored in the EEPROM 406 or the like, and is 
checked when determining a reporting destination to be 
described later. In other words, the reporting 

15 destination during a reporting event occurrence is 

controlled by checking this table. It should be noted 
that, when a person other than one previously 
registered is selected, then processing proceeds to 
registration of the person who is the object of 

20 recognition (905) from a new registration screen (not 
shown). In the registration process (905), video of 
the person to be registered is imaged and the feature 
parameters necessary to recognize such registered 
person is extracted from this video data. Furthermore, 

25 in the registration process (905), the user is prompted 
to enter attribute information for the registered 
person (such as name, etc.). 
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FIG. 25 shows a sample reporting control 
information table showing the relation between a person 
who is the object of recognition, the content of the 
reporting and the reporting destination. The location 
5 code is a unique code assigned to the location 

recognized in the place of installation recognition 
step S2202. The person code is a unique code assigned 
to previously registered persons. 

It should be noted that it is also possible to 

10 establish a plurality of persons as the object of 

recognition for a location (as in the case of location 
code P0002 shown in FIG. 25). In this case, an order 
of priority of the objects of recognition may be added 
to the reporting control information table. If an 

15 order of priority is established, then in a process of 
analyzing the content of the situation (step S2205) the 
situation of a person of higher priority is subjected 
to recognition processing more frequently. Furthermore, 
sometimes a particular person who is an object of 

20 recognition is not set for a given location (as in the 
case of location code P0004 in FIG. 25). In this case, 
when a predetermined situation at that location is 
recognized (such as intrusion by a person), the 
reporting process is executed in step S2209 regardless 

25 of the output of the object recognition process of step 
S2206. 

Next, in step S2403, the content of the 
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situation for wliich reporting is to be carried out is 
set for each person who is the object of recognition. 
FIG. 26 shows one example of display contents displayed 
on the LCD 2301 of the . controls 409. When buttons 504- 
5 505 are pressed, previously registered recognition 
situation contents are displayed in succession. When 
button 506 is pressed, the situation currently 
displayed is set as the reporting occurrence situation 
for that person who is the object of recognition object 

10 of recognition. 

When selection of the situation content is 
completed and the OK button 507 is pressed, the 
situation content at the current place of installation 
is set in the reporting control infoxmation table (FIG. 

15 25). It should be noted that when the "default" (2602) 
is set or when there is no input from the user for a 
predetermined period of time, the content is 
automatically set to the default setting. The default 
is such that a situation ordinarily set in most cases, 

20 such as recognition of "room entry and exit" and the 
like, is automatically designated, thereby eliminating 
the inconvenience attendant upon setting. 

Next, in step S2404, the reporting destination 
for the reporting is set for each object of recognition 

25 and its situation content. FIG. 27 shows a sample 
display of a reporting destination setting screen 
displayed on the LCD 2301 of the controls 409. When 
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buttons 504-505 are pressed, previously registered 
reporting destinations are displayed in succession. 
When button 506 is pressed, the reporting destination 
currently displayed is set as the reporting destination 
5 when a situation of the person who is object of 
recognition is recognized. 

When selection of the situation to be recognized 
is completed and the OK button 507 is pressed, the 
reporting destination is set in the reporting control 

10 information table (PIG. 25). It should be noted that, 
if a "new registration" (2705) is set, then a 
predetermined interface, not shown, is displayed on the 
predetermined control panel 501 and registration of a 
new reporting destination is carried out. In addition, 

15 it is also possible to set a plurality of reporting 
destinations for a single situation. 

As described above, in steps S2402-S2404 the 
reporting control information table (FIG. 25) for a 
given location is set . To explain in specific terms 

20 using FIG. 25, if the location code is P0002, the query 
"Has person fallen?" is set as the reporting condition 
for person HlOOl and a report to that effect is made to 
"Father" if that condition is recognized. 

In addition, the queries "Has person put 

25 something in his mouth" and "Is person in a prohibited 
axea?" are set as reporting conditions for person H1002, 
and reports are made to that effect to "Mother" and 
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"Older Brother" if situations of such Gona.itions are 
recognized. It should be noted that in the case of 
locations for which no particular persons are 
registered, the system recognizes the situations of all 
5 persons or the situation of that location (such as the 
outbreak of a fire and so forth). For example, in FIG, 
25, at location P0004, such recognition processes as 
detection of the entry of all persons or detection of a 
suspicious person are executed and a report to that 

10 effect is made to "Security Company" if intrusion by a 
person is detected. 

As described above, in step S2203, the object of 
recognition, the situation to be recognized and the 
corresponding reporting destination are recorded in the 

15 reporting control information table. 

Next,, in step S2204., it is determined whether or 
not there has been a change in situation. Here, for 
example, using the difference between frames of image • 
data, the system detects changes in image in the area 

20 of the object of recognition. If a change beyond a 
predetermined area is confirmed in this step, then in 
step S2205 the process of analyzing the content of the 
situation of the target object is commenced. It should 
be noted that, in step S2204, for example, a change in 

25 situation may be detected using information other than 
image data. For example, a technique may be used in 
which intrusion by a person is detected using a sensor 
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that uses Infrared rays or the like. In this step, a 
change in the situation (such as the presence of a 
person) is detected with a simple process and the 
process of analyzing the content of the situation (step 
5 S2205) is executed only when necessary. 

Wien a change in situation is detected, in step 
S2205 the process of analyzing the change in situation 
is executed. In step S2205, a person within the 
sensing range is tracked and the situation of that 

10 person is analyzed. It should be noted that it is 
possible to utilize any of the variety of methods 
proposed conventionally for the necessary situation 
recognition technique. For example, detection of the 
entry into a room of a particular person or the entry 

15 of a suspicious person into the room can be 

accomplished easily using individual identification 
results produced by face detection/face recognition 
techniques. In addition, many techniques for 
recognizing facial expression have been proposed, such 

20 as the device proposed by Japanese Laid-Open Patent 
Publication No. 11-214316 that recognizes such 
expressions as pain, excitement and so forth. 

Furthermore, a situation in which an infant has 
put a foreign object into his or her mouth also can be 

25 recognized from recognition of hand movements proposed 
in conventional sign language recognition and the like 
and from information concerning the position of the 
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mouth obtained by detection of the face. Furthermore, 
in Japanese Laid-Open Patent Publication No. 6-251159, 
a device that converts feature vector sequences 
obtained from time series images into symbol sequences 
5 and selects the most plausible from among the object of 
recognition categories based on a hidden Markov model 
is proposed* 

In addition, in Japanese Laid-Open Patent 
Publication No- 01-268570, a method of recognizing a 

10 fire from image data is proposed. In step S2205, 
processing modules including this plurality of 
situation recognition algorithms are executed, the 
output values of the processes are determined and 
whether or not a predetermined situation has occurred 

15 is output. 

FIG. 35 is a diagram showing one example of a 
recognition processing software module provided in step 
S2205. Reference numerals 3501-3505 correspond to a 
module for recognizing the posture of a person, a 

20 module for detecting an intruder in a predetermined 

area, a module for recognizing a person's expressions, 
a module for recognizing predetermined movements of a 
person, and a module for recognizing environmental 
situations (that is, recognition of particular 

25 situations such as a fire or the like) , respectively, 
which process image data imaged by the video input unit 
410 (and stored in the RAM 405). 
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The modules operate as middle ware tasks either 
by time division or serially. In this step, the output 
values of the modules are output as the results of 
analysis of data encoded into a predetermined format* 
5 It should be noted that these modules may also be 

implemented as special hardware modules. In that case, 
the hardware modules are connected to the system bus 
404 and process the image data stored in the RAM 405 at 
a predetermined time. 

10 In step S2206, the person who is the object of 

recognition of the situation recognized in the process 
of analyzing the content of the situation (step S2205) 
is recognized. Any of the variety of techniques 
proposed conventionally can be adapted to that 

15 processing relating to recognition of the person which 
is necessary to this step (e.g., S. Akamatsu: 
"Research Trends in Pace Recognition by Computer" , 
Transactions of the Institute of Electronics, 
Information and Communication Engineers, vol. 80 No. 3, 

20 pp. 257-266 (March 1997)). It should be noted that the 
feature parameters needed to identify an individual are 
extracted during new registration of the individual as 
described above (reference numeral 905 shown in FIG. 9). 
In step S2207, the reporting control information 

25 table is checked and it is determined whether or not a, 
predetermined situation of a predetermined person which 
should be reported has been recognized, and if so, in 
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step S2208 the process of encoding the content of the 
situation is carried out. It should be noted that 
although in FIG. 25 the description of the situation 
content that is reported is shown as words expressing a 
5 predetermined situation, in actuality a code 

corresponding to predetermined code data, not shown, 
that the process of analyzing the content of the 
situation (step S2206) outputs {that is, a code 
uniquely specifying a corresponding situation) is 

10 recorded in the table. 

Next, a process of encoding the content of the 
situation (step S2208) converts the situation content 
into predetermined character information using the 
output from the process of analyzing the content of the 

15 situation (step S2206}. This conversion may, for 
example, provide a conversion table determined in 
advance, with the character information obtained from 
the output of the process of analyzing the content of - 
the situation (step S2206) and the content of such 

20 table • 

FIG. 28 is a diagram showing a sample conversion 
table. For example, a situation recognition processing 
module ROOOl (corresponding to the recognition module 
3501 shown in FIG. 35), recognizes and outputs three 
25 types of situations for a person. A situation 

recognition processing module R0003 (corresponding to 
the recognition module 3503 shown in FIG. 35), 
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recognizes and outputs two types of situations for a 
person. If a predetermined output is obtained from the 
recognition processing modules (reference numerals 
3501-3505 shown in FIG. 35), the conversion table Is 
5 checked and the corresponding predetermined character 
sequence is output. Thus the process of encoding the 
content of the situation {step S2208), using the output 
values (predetermined codes) of the process of 
analyzing the content of the situation (step S2205), 

ID obtains character information by checking the 
conversion table. It should be noted that the 
conversion table is assumed to be recorded in advance 
in the EEPROM 406. 

iFIG, 36 shows details of the reporting process 

15 (step S2209). In this step, the person to be notified 
is determined on the basis of the output of the process 
of identifying the place of installation (step S2202), 
the process of analyzing the content of the situation 
(step S2205) and the process of identifying the object 

20 of recognition (step S2206), and by checking the 

reporting control information table (FIG. 25) stored in 
the EEPROM 406 in step S3601. 

Next, in step S3602, the character information 
obtained in the situation encoding process (step S2208) 

25 is transmitted to the person to be notified. The 
character information is transmitted via the 
communications interface 408 in accordance with a 



wo 2005/119620 



65 



PCT/JP2005/010724 



protocol such as electronic mail, instant messaging or 
the like. It should be noted that the selection of the 
reporting destination, in the case of e-mail, is 
accomplished by establishing a particular e-mail 
5 address for the reporting destination. 

It should be noted that, after power is supplied 
to the main unit, the processes of steps S2204-S2209 
are executed repeatedly, and when a predetermined 
situation is recognized, the content of the situation 
10 is reported to the person to be notified in that 
situation. 

As can be understood from the foregoing 
description, according to the present embodiment, when 
a predetermined situation is recognized the content of 

15 that situation can be easily grasped, and furthermore, 
the appropriate reporting destination can be notified 
of the content of that situation depending on the place 
of installation of the device, the object of 
recognition and the situation to be recognized* 

20 (Fifth Embodiment) 

FIG. 30 is a diagram showing the structure of a 
situation monitoring device according to a fifth 
embodiment of the present invention. The hardware 
configuration of this embodiment differs from that of 

25 the first embodiment shown in FIG, 4 only insofar as 
the communications interface 408 is different. 

Reference numeral 3001 designates a CPU. 
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Reference numeral 3302 designates a bridge, wliich has 
the capability to bridge a high-speed CPU bus 3003 and 
a low-speed system bus 3004. 

In. addition, the bridge 3002 has a built-in 
5 memory controller function, and thus the capability to 
control access to a RAM 3005 connected to the bridge. 
The RAM 3005 is the memory necessary for the operation 
of the CPU 3001, and is composed of large -capacity, 
high-speed memory such as SDRAM/DDR/RDRAM and the like. 

10 In addition, the RAM 3005 is also used as an image data 
buffer and the like. 

Furthermore, the bridge 3002 has a built-in DMA 
function that controls data transfer between devices 
connected to the system bus 3004 and the RAM 3005. An 

15 EEPROM 3006 is a memory for storing the instruction 
data and a variety setting data necessary for the 
operation of the CPU 3001. 

Reference numeral 3007 designates an RTC IC, 
which is a special device for carrying out time 

20 management /calendar management. Reference numeral 3009 
designate the controls, and is a processor that 
controls the user interface between the main unit and 
the user. The controls 3009 are incorporated in a rear 
surface or the like of a stand 304 of the main unit. 

25 Reference numeral 3010 designates a video input unit, 
and includes photoelectric conversion devices such as 
CCD /CMOS sensors as well as the driver circuitry to 
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control such, devices, the signal processing circuitry 
to control a variety of image corrections, and the 
electrical and mechanical structures for implementing 
pan/tilt mechanisms. 
5 Reference numeral 3011 designates a video input 

interface, which converts raster image data output from 
the video input unit 3010 together with a sync signal 
into digital image data and buffers it. In addition, 
video input interface 3011 has the capability to 

10 generate signals for controlling the video input unit 
3010 pan/ tilt mechanism- The digital image data 
buffered by the video input interface 3011 is, for 
example, forwarded to the predetermined address in the 
RAM 3005 using the DMA built into the bridge 3002. 

15 Such DMA transfer may, for example, be activated 

using the video signal vertical sync signal as a 
trigger. The CPU 3001 then commences processing the 
image data held in the RAM 3005 based on a DMA 
transfer- completed interrupt signal that the bridge 

20 3002 generates. It should be noted that the situation 
monitoring device also has a power supply, not shown. 

Reference niimeral 3008a designates a first 
communications interface, having the capability to 
connect to a wireless /wire LAN internet protocol 

25 network. Reference numeral 3008b has the capability to 
connect directly to an existing telephone network or 
mobile telephone network. In the present embodiment. 
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the reporting medltim Is selected according to the 
object to be recognized and the situation thereof. 
Specifically, when reporting a normal situation, 
depending on the degree of urgency the information is 
5 reported using an internet protocol such as electronic 
mail. Instant messaging or the like. If the situation 
is an urgent one, then the situation content is 
reported directly by telephone or the like. 

FIG. 31 is a flow chart illustrating details of 

10 the reporting destination setting process (step S2203} 
according to the present embodiment. In this 
embodiment, compared to the fourth embodiment described 
above a new reporting medium setting process (step 
S3105) is added. The other steps S3101-S3104 are the 

16 same as steps S2401-S2404 described in the fourth 
embodiment, and a description thereof is omitted. 

FIGp 32 is a diagram showing the content of the 
reporting control information table used in the present 
embodiment. In the reporting medium setting process 

20 (step S3105), the reporting medivrai is set according to 
the place of recognition, the object of recognition and 
the content of the situation. In the case of PIG. 32, 
it is specified that reporting is to be "by telephone" 
for such extremely urgent situations as "Has person 

25 fallen?" and "Suspicious person detected". By 

contrast, "by instant messaging" is specif led for such 
situations of Intermediate urgency as "Is person in 
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pain?", ""Has person put something in his mouth?" and 
"Is person in a prohibited area?", and "by e-mail" is 
specified for such situations of lesser urgency as 
"Entry/exit confirmed". 
5 The information set in step S3105, as with the 

fourth embodiment described above, is then recorded in 
the EEPROM 3005 as the reporting control information 
table . 

In the situation content encoding process (step 
10 S2208) of the present embodiment, the situation content 
is encoded according to the reporting medium set in the 
reporting medium setting process (step S2203), For 
example, character information is encoded if "instant 
messaging" or "e-mail" are set as the reporting medium, 
15 and voice information is encoded if "telephone" is set 
as the reporting medium. The encoding of voice 
information generates voice data corresponding to the 
character sequence shown in the table shown in FIG. 28- 
by a voice synthesis process, not shown. It should be 
20 noted that such voice data may be compressed using 
high" efficiency compression protocols such as ITU 
standard G.723 or G.729. The voice information thus 
generated is then temporarily stored in the RAM 3005 or 
the like. 

25 FIG. 37 is a diagram illustrating details of the 

reporting process (S2209). In step S3701, the 
reporting control information table (FIG. 32) stored in 
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the EEPROM 3006 is checked and a predetermined 
reporting destination is determined according to the 
output of the process of identifying the place of 
installation (step 82202), the putput of the process of 
5 identifying the object of recognition (step S2206) and 
the output of the process of analyzing the content of 
the situation (step S2205). 

Next, in step S3702, similarly, the reporting 
control infoxiaation table is checked and the reporting 

10 medium determined. Encoded information expressing the 
content of the situation is then transmitted to the 
reporting destination selected in step S3 702 through 
the selected reporting medium (3008a or 3008b). In 
other words, if '"instant messaging", ''e-mail" or the 

15 like is selected as the reporting medium, the report 
content is transmitted according to internet protocol 
through the first communications interface 3008a. If 
"telephone" is selected as the reporting medium, then 
the telephone of the predetermined reporting 

20 destination is automatically called and after ringing 
is confirmed the voice data held in the RAM 3005 is 
transmitted as direct audio signals through the second 
communications interface 3008b. 

Thus, according to the present embodiment, it is 

25 possible to notify a predetermined reporting 

destination by reporting medium selected according to 
the situation, achiiaving a reporting capability suited 
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to the degree of urgency. 

(Sixth Embodiment) 

FIG, 33 is a diagram showing the outlines of a 
processing flow . performed by a situation monitoring 
5 device according to a sixth embodiment of the present 
invention. The flow chart is a program loaded in the 
RAM 3005 and processed by the CPU 3001. The hardware 
configuration of the situation monitoring device 
according to the present embodiment is the same as that 

10 of the fifth embodiment, and therefore a description is 
given only of the difference between the two. 

FIG. 33 is a flow chart illustrating details of 
the reporting destination setting process (step S2203) 
of the present embodiment. In this embodiment, in 

15 contrast to the reporting destination setting process 
of the fifth embodiment, a reporting determination time 
setting process step (S3306) is newly added. The 
remaining steps S3301-S3305 are each the same as steps 
3101-S3105 described in the fourth embodiment, and thus 

20 a description of only the difference therebetween is 
given . 

FIG. 34 is a diagram showing one example of a 
reporting control information table according to the 
present embodiment. In the event that time information 
25 corresponding to recognition situations is set and a 
predetermined situation is recognized, that recognized 
time is determined and the content of the recognition 
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situation is reported to the reporting destination in 
accordance with the time. For example, in the case of 
location code P0003, if an intruder is detected between 
the hours of 0800 and 2400, the system is set to notify 
5 the mother by electronic mail. By contrast, if an 

intruder is detected between the hours of 2400 and 800 
under the same conditions, the system is set to notify 
the security company. The information set in step 
S3306, as with the fourth embodiment, is recorded in 
10 the EEPROM 3006 as a reporting control information 
table . 

FIG. 38 is a flow chart illustrating details of 
the reporting process (step S2209) according to the 
present embodiment. In step S3801, the time that a 

15 predetermined situation is recognized is obtained from 
the RTC 3007. In step S3802, based on the place of 
recognition, the person who is the object of 
recognition, the recognition situation and the time 
obtained in step S3801, the reporting control 

20 information table {FIG. 34) stored in the EEPROM 3006 
is checked and a predetermined reporting destination 
determined • 

Furthermore, in step S3803, the reporting 
control information table is similarly checked and a 

25 predetermined reporting medium determined. In step 

S3804, data encoded in step S2208 showing the content 
of the situation is transmitted to the reporting 
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destination determined in step S3803 through reporting 
medium determined in step S3 80 4 . 

As can be understood from the foregoing 
description, with the present embodiment, based on the 
5 time when a predetermined situation is recognized, it 
is possible to report to more appropriate reporting 
destinations using more appropriate reporting medium. 

It should be noted that although the foregoing 
embodiments are described in terms of a person as the 

10 object of recognition, the present invention is not 

limited thereto and the object of recognition may be an 
animal, a particular object or anything else. For 
example, in the case of a particular object, situations 
such as that object "^Has been moved from a 

15 predetermined position" or "Has gone missing" may be 
recognized and reported. The recognition of movement 
or presence/absence can be easily accomplished by the 
use of pattern matching techniques proposed 
conventionally . 

20 Although in the foregoing embodiments the 

reporting control information table specifies the 
reporting destination and reporting medium depending on 
the place of installation of the device and the object 
of recognition, the time and the situation, the present 

25 invention is not limited thereto. Depending on the 
purpose, a table that designates the reporting 
destination or the reporting medium according to at 
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least one of the place of Installation, the object of 
recognition and the time as well as the situation may 
be provided. 

Although the foregoing embodiments are described 
5 in terms of the process of analyzing the content of the 
situation by providing a plurality of situation 
recognition processes and utilizing the output of those 
processes to analyze the situation content, the present 
invention is not limited thereto and any method may be 
10 used. For example, a more generalized recognition 

algorithm may be installed and all target situations 
recognized. 

Although the foregoing embodiments are described 
in terms of encoding the results of the process of 

15 analyzing the content of the situation as predetermined 
character sequences or audio information, the present 
invention is not limited thereto and these results may 
be converted into other types of information. For 
example, such information may be converted into 

20 diagrammatic data that expresses the information 

schematically, and such diagrammatic data transmitted 
as reporting data. In addition, instead of reporting 
over a network, a method may be used in which light 
patterns from a predetermined light source are reported 

25 as warning information. 

Although the fourth embodiment described above 
is described in terms of using video information to 
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recognize the place of installation of the device and 
the situation of the object of recognition, the present 
invention is not limited thereto and sensing 
information other than video information may be used to 

5 recognize the situation. Furthermore, situations may 
be recognized using a combination of video information 
and other sensing information. As other sensing 
information it is possible to use a variety of sensing 
technologies such as audio information, infrared ray 

10 information and electromagnetic information. 

Although the foregoing embodiments are described 
in terms of the medium that report a change in the 
situation of the object of recognition as internet mail, 
instant messaging and telephone, etc., the present 

15 invention is not limited thereto and other medium may 
be used as necessary. 

Although the foregoing embodiments are described 
in terms of establishing the reporting control 
information table using the controls 409, alternatively 

20 a network may be used to set the parameters necessary 
for operation. In this case, the main unit may have a 
HTTP (Hyper Text Transfer Protocol) searver capability, 
for example, and provide a Web-based user interface to 
the user through the communications interface 3008. 

25 The HTTP server is incorporated as one type of middle 
ware, and activates a predetermined parameter setting 
program in response to operation from a remote location 
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based on HTTP. 

In this case, the user can set the parameters 
necessary for operation of the main unit from an 
ordinary terminal such as a mobile telephone, a PDA or 
5 a personal computer, and furthermore, such setting 
operations can be carried out from a remote location. 

Although the foregoing embodiments are described 
in terms of executing all processing such as the 
recognition processing using a processor built into the 

10 main unit, the present invention may be implemented, 

for exeunple, in combination with an external processing 
device such as a personal computer or the like. In 
this case, only the reading in of image data is 
accomplished using a specialized device, with the 

15 remaining processes, such as image recognition and 
communications, implemented using personal computer 
resources . 

By using a wireless interface such as BlueTOoth,- 
for example, or a power line communications interface 

20 such as HPA (Home Power Plug Alliance) or the like to 
connect the specialized device and the personal 
computer, the same convenience can be achieved. This 
sort of functionally dispersed situation monitoring 
system can of course be achieved not only with the use 

25 of a personal computer but also with the aid of a 
variety of other internet appliances as well. 

Although the foregoing embodiments are described 
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in terms of implementing the present invention by 
software processing using a CPU, the present invention 
is not limited thereto and may, for example, be 
implemented by special hardware processing as well* In 
5 that case, the algorithm for situation recognition 

corresponds to object data that determines the internal 
circuitry of an FPGA (Filed Progranmiable Gate Array) or 
object data that determines the internal circuitry of a 
reconf igurable processor. The system control processor 

10 loads the data from the EEPROM 406 or a server device 
connected to the network and the like into the special 
hardware. The special hardware then commences 
recognition processing of a predetermined algorithm 
according to the object data that has been loaded. 

15 Although the foregoing embodiments are described 

in teirms of using a camera having a mechanical control 
structure (a so-called pan/tilt camera), the present 
invention is not limited thereto and may, for example • 
employ a wide-angle camera instead. In that case, the 

20 object of recognition is not supplemented mechanically 
but instead an equivalent process can be implemented 
using image data acquired at wide angles « 
(Other Embodiments) 

It should be noted that the present invention 
25 can be adapted to a system comprised of a plurality of 
devices (for example, a host computer, an interface 
device, a reader, a printer and so forth) or to an 
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apparatus comprised of a single device. 

In addition, the invention can be implemented by 
supplying a software program, which implements the 
functions of the foregoing embodiments, directly or 

5 indirectly, to a system or apparatus, reading the 

supplied program code with a computer (or CPU or MPU) 
of the system or apparatus, and then executing the 
program code. 

In this case, the functions of the foregoing 

10 embodiments are implemented by the program code itself 
read from the storage medium, and the storage medium 
storing the program code constitutes the invention. 

Examples of storage media that can be used for 
supplying the program code are a floppy disk 

15 (registered trademark), a hard disk, an optical disk, a 
magneto -optical disk, a CD-ROM, a CD-R, magnetic tape, 
a nonvolatile type memory card, a ROM or the like. 

Besides those cases in which the aforementioned 
functions according to the embodiments are implemented 

20 by executing the program code read by computer, the 
present invention also includes a case in which an OS 
(operating system) or the like running on the computer 
perfoanns all or part of the actual processing according 
to the program code instructions, so that the functions 

25 of the foregoing embodiments are implemented by this 
processing. 

Furthermore, after the program read from the 
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storage medium is written to a function expansion board 
inserted into the computer or to a memory provided in a 
function expansion unit connected to the computer, a 
CPU or the like mounted on the function expansion board 
5 or function expansion unit performs all or part of the 
actual processing so that the functions of the 
foregoing embodiment can be implemented by this 
processing. 

The present invention is not limited to the 
10 above embodiments and various changes and modifications 
can be made within the spirit and scope if the present 
invention. Therefore, to apprise the public of the 
scope of the present invention, the following claims 
are made. 

15 

CLAIM OF PRIORITY 

This application claims priority from Japanese 
Patent Application No. 2004-167544 filed on June 4, 
2004 and Japanese Patent Application No. 2005-164875 
20 filed on June 3, 2005, the entire contents of which are 
hereby incorporated by reference herein. 
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CLAIMS 

1. A situation monitoring device comprising: 
place recognition means for recognizing a place 

5 of installation where the device is installed; 

information holding means for holding relational 
information relating the place of installation and a 
situation to be recognized; 

determination means for deteannining a 
10 predetermined situation to be recognized, in accordance 
with recognition results by the place recognition means 
and the relational information; 

situation recognition means for recognizing the 
predetermined situation determined by the determination 
15 means; and 

communications means for reporting the 
recognition result of the predetermined situation 
recognized by the situation recognition means to the 
user. 

20 

2. A situation monitoring device comprising: 
place recognition means for recognizing a place 

of installation where the device is installed; 

target object recognition means for recognizing 
25 the target object; 

information holding means for holding relational 
information relating the place of installation with the 
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tcirget object to be recognized and a situation to be 
recognized; 

determination means for determining the target 
object to be recognized and a predetermined situation 
5 to be recognized, in accordance with recognition 
results by the place recognition means and the 
relational information; 

situation recognition means for recognizing that 
the target object determined by the determination means 
10 is in the predetermined situation determined by the 
determination means? and 

communications means for reporting the 
recognition result of the predetermined situation 
recognized by the situation recognition means to the 
15 user. 



3. A situation monitoring device comprising: 

target object recognition means for recognizing- 
a particular target object; 
20 information holding means for holding relational 

information relating the target object and a situation 
to be recognized; 

determination means for determining a 
predetermined situation to be recognized, in accordance 
25 with recognition results by the target object 

recognition means and the relational information; 

situation recognition means for recognizing that 
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the target object recognized by the target object 
recognition means is in the predetermined situation 
determined by the determination means; and 

communications means for reporting the 
5 recognition result of the predetermined situation 
recognized by the situation recognition means to the 
user. 

4. The situation monitoring device according 
10 to any one of claims 1 through 3, wherein the situation 
recognition means comprises an acquisition means for 
acquiring image data, and recognizes the predetermined 
situation from the acquired image data. 

15 5. The situation monitoring device according 

to either of claims 1 and 2, wherein the place of 
Installation recognition means comprises an acquisition 
means for acquiring image data, and recognizes the 
place of installation from the acquired image data. 

20 

6. The situation monitoring device according 
to either of claims 1 and 2, wherein the place 
recognition means commences a process of recognition of 
the place of installation when a predetermined 
25 condition is detected. 



7. The situation monitoring device according 
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to Claim 6, wherein the predetermined condition is a 
change in the sensed image. 

8- The situation monitoring device according 
5 to claim 6, wherein the place recognition means 

comprises a sensor for detecting movement of the device 
and the predetermined condition is a change in such 
sensor information. 

10 9- The situation monitoring device according 

to claim 6, further comprising controls for inputting 
parameters necessary for operation of the device, and 
the predetermined condition is a particular input by 
the user to the controls. 

15 

10. The situation monitoring device according 
to claim 6, wherein the predetermined condition is 
power on of the device - 

20 11. The situation monitoring device according 

to claim 6, wherein the predetermined condition is a 
time determined in advance. 

12. The situation monitoring device according 
25 to any one of claims 1, 2, and 4 through 11, wherein 
the communications means further reports to the user 
that a shift in the place of installation has been 
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recognized by the place recognition means. 

13. The situation monitoring device according 
to any one of claims 1 through 12, wherein the 

5 communications means further reports to the user that 
the object to be recognized has changed. 

14. The situation monitoring device according 
to any one of claims 1 through 5, further comprising 

10 controls for inputting parameters necessary for 

operation of the device and an interface prompting the 
user to update the relational information under 
predetermined conditions is displayed on the controls. 

15 15. The situation monitoring device according 

to claim 14, wherein the predetermined condition is the 
place recognition means recognizing a shift in the 
place of installation - 

20 16. The situation monitoring device according 

to claim 14, wherein the predetermined condition is the 
place recognition means recognizing a place of 
installation that is not registered in the relational 
information . 

25 

17. The situation monitoring device according 
to claim 14, wherein the predetermined condition is the 
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target object recognition means recognizing a target 
object that is not registered in the relational 
information • 

5 18. The situation monitoring device according 

to any one of claims 1, 2, and 4 through 17, wherein a 
situation of a default determined in advance is 
determined by the determination means when the place 
recognition means recognizes a place of installation 

10 that is not registered in the relational information, 

19. The situation monitoring device according 
to any one of claims 3, 4, and 13 through 17, wherein a 
situation of a default determined in advance is 

15 determined by the determination means when the target 
object recognition means recognizes a target object 
that is not registered in the relational information. 

20. The situation monitoring device according 
20 to any one of claims 1-19, wherein the situation 

recognition means recognizes a situation in accordance 
with an order of priority determined in advance when a 
plurality of target objects exist for a recognized 
location. 

25 

21. The situation monitoring device according 
to either of claims 1 and 2, wherein the device has a 
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configuration dispersed In a main part and a peripheral 
part, and information for recognizing the place of 
installation with the place recognition means is held 
in the peripheral part. 

5 

22. The situation monitoring device according 
to either of claims 1 and 2, wherein the place 
recognition means further comprises external 
communications means for communicating with an external 
10 device disposed adjacent to an external apparatus or 

the main unit, and recognizes the place of installation 
according to information emitted by the external 
apparatus or information held by the external device. 

15 23. The situation monitoring device according 

to any one of claims 1 through 22, further comprising 
controls separate from the main unit and controls 
communications means for communicating with the 
controls, wherein the setting of parameters necessary 

20 for operation of the device is carried out using the 
controls . 

24. A situation monitoring device according to 
any one of claims 1 through 23, further comprising 
25 connection means for connecting to a network and a 
server device, wherein the setting of parameters 
necessary for operation of the device is carried out 
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from the external apparatus using the server device. 



25. The situation monitoring device according 
to claim 24, wherein the server device is a HTTP (Hyper 
5 Text Transfer Protocol) server. 



26. A situation monitoring system comprising: 
the situation monitoring device according to any 

one of claims 1-25; and 
10 connection means for connecting to a network, 

wherein a processing algorithm executed by the 

situation recognition means is held in another external 

apparatus connected to the network. 

15 27. A situation monitoring device comprising: 

situation analyzing means for analyzing a 
situation of a target object; 

discrimination means for identifying a 
predetermined situation from output from the situation 
20 analysis means; 

situation encoding means configured to convert 
the situation into a predetermined signal based on the 
output from the situation analysis means; and 

communications means for reporting the output of 
25 the situation analysis means to the user using the 
situation encoding means. 
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28. The situation monitoring device according 
to claim 27, wherein the predetermined signal is at 
least one of diagrammatic symbol information, character 
information, voice information and light -emitting 
5 pattern inf oaimation expressing the situation content, 
encoded by a predetermined method. 



29. The situation monitoring device according 
to claim 28, further comprising control means for 

10 selecting at least one reporting destination from among 
predetermined reporting destinations based on the 
output of the situation analysis means. 

30. The situation monitoring device according 
15 to claim 29, further comprising target object 

identification means for identifying a target object, 
wherein the control means selects at least one 
reporting destination from among predetermined 
reporting destinations based on output of the target 
20 object identification means and the output of the 
situation analysis means for that target object. 



31. The situation monitoring device according 
to claim 29, further comprising location identification 
25 means for identifying a location, 

wherein the control means selects at least one 
reporting destination from among predetermined 
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reportirxg destinations based on output of the location 
Identification means and the output of the situation 
analysis means. 

5 32. The situation monitoring device according 

to claim 29, further comprising: 

target object identification means for 
identifying a target object? and 

location identification means for identifying a 
10 location, 

wherein the control means selects at least one 
reporting destination from among predetermined 
reporting destinations based on the output of the 
target object identification means and the output of 
15 the location identification means as well as the output 
of the situation analysis means. 

33. The situation monitoring device according 
to any one of claims 29 through 32, further comprising 
20 time management means for managing time, 

wherein the control means selects at least one 
reporting destination from eunong predetermined 
reporting destinations based on output of the time 
management means and the output of the situation 
25 analysis means. 



34. The situation monitoring device according 
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to either of claims 27 or 28, comprising a plurality of 
communications means and control means for selecting at 
least one communications means from among the plurality 
of communications means based on the output of the 
5 situation analysis means. 

35. The situation monitoring device according 
to claim 3A, further comprising target object 
identification means for identifying a target object, 

10 wherein the control means selects at least one 

communications means from among the plurality of 
communications means based on output of the target 
object identification means and the output of the 
situation analysis means for the target object. 

15 

36. The situation monitoring device according 
to claim 34, further comprising location identification 
means for identifying a location, 

wherein the control means selects at least one 
20 communications means from among the plurality of 

communications means based on output of the location 
identification means and the output of the situation 
analysis means. 



25 37. The situation monitoring device according 

to claim 34, further comprising: 

target object identification means for 
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identifying a target object; and 

location identification means for identifying a 
location , 

wherein the control means selects at least one 
5 communications means from among the plurality of 

communications means based on the output of the target 
object identification means, the output of the location 
identification means and the output of the situation 
analysis means. 

10 

38. The situation monitoring device according 
to any one of claims 34 through 37, further comprising 
a time management means for managing time, 

wherein the control means selects at least one 
15 reporting destination from among predeteznxiined 

reporting destinations based on output of the time 
management means and the output of the situation 
analysis means. 

20 39- The situation monitoring device according 

to any one of claims 27 through 38, wherein the 
* situation cinalysis means comprises image data 
acquisition means for acquiring image data, and 
analyzes the situation from the image data acquired by 

25 the image data acquisition means. 



40. The situation monitoring device according 
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to any one of claims 30, 32, 35 and 37, wherein the 
target object identification means for Identifying a 
target object comprises image data acquisition means 
for acquiring image data, and identifies the target 
5 object using the image data acquired by the image data 
acquisition means. 

41. The situation monitoring device according 
to any one of claims 31, 32, 36 and 37, wherein the 

10 location identification means for identifying a 

location comprises image data acquisition means for 
acquiring image data, and identifies the location using 
the image data acquired by the image data acquisition 
means. 

15 

42. The situation monitoring device according 
to any one of claims 27 through 41, further comprising: 

connecting means for connecting to a network; 

and 

20 a server function, 

wherein parameters necessary for operation of 
the device can be set from an external terminal device 
through the server function. 

25 43. The situation monitoring device according 

to claim 42, wherein the parameters are control 
parameters for the control means that selects the 
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reporting destination. 

44- Tlie situation monitoring device according 
to claim 42, wherein the parameters are control 
5 parameters for the control means that selects the 
reporting means. 

45. The situation monitoring device according to any 
one of claims 42 through 44, wherein a HTTP (Hyper Text 
10 Transfer Protocol) server implements the server 
function . 
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